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ABSTRACT 

Several models for optimizing incomplete sample 
designs with respect to information on the item parameters are 
presented. The following cases are considered: (l) known ability 
parameters; (2) unknown ability parameters; (3) item sets with 
multiple ability scales; and (4) response models with multiple item 
parameters. The models are able to cope with hierarchical structures 
in the population of examinees as well as the domain of content, and 
allow for practical constraints with respect to such items as test 
content, curricular differences between groups, or time available for 
item administration. An example with test data from a national 
assessment study illustrates the use of the models. This methodology 
was applied to an imagined third study of the Dutch part of the 
Second Mathematics Study of the International Association for the 
Evaluation of Educational Achievement for the three subject areas of 
Geometry, Algebra, and Arithmetic for a sample of 400 seventh 
graders. The LANDO computer program was used to solve the models, 
illustrating their utility. (Author/SLD) 
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Abstract 

Several models for optimizing incomplete sample designs with 
respect to information on the item parameters are presented. 
The following cases are considered: (1) known ability 
parameters; (2) unknown ability parameters; (3) item sets 
with multiple ability scales; and (4) response models with 
multiple item parameters. The models are able to cope with 
hierarchical structures in the population of examinees as 
well as the domain of content, and allow for practical 
constraints with respect to f e.g., test content, curricular 
differences between groups, or time available for item 
administration. An example with test data from a national 
assessment study illustrates the use of the models. 

Keywords- Item Response Theory. Maximum-likelihood Estima- 
tion; Incomplete Sample; Optimal Experiment; Linear 
Programming 
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Optimizing Incomplete Sample Designs for Estimating 
Item Response Model Parameters 

It is well known in item response theory that the accuracy of 
the maximum-likelihood estimators of the item parameters 
depends on the abilities of the examinees in the sample. This 
fact can easily be demonstrated for the Rasch model. Suppose 

item i = 1 I can be characterized by a scalar parameter 

8^ representing its difficulty, whereas the scalar parameter 

Gj measures the ability of examinee j = 1 J to solve 

the items. For stochastic responses Aj^ € {0, 1}, the model 
gives the probability of event {Aj^ = 1} as 



with —o < 5 i( 6j < +oo (Pasch. 1980). If j = 1, . ... J and 
independent responses from all the examinees are used to 
estimate 8^., then for J — > «> the asymptotic variance of the 
maximum-likelihood estimator is equal to the reciprocal of 
Fisher's information measure 



(1) 



P(A i5 = 1) = [1 + expa L - 




-1 



(2) 




where p^Qj) s P(Aij = 1) . 
Obviously. 
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(3) 



max 



..ej 



1(51:6! 6j) = I(5 i ;e 1 V..,e J , 5 



with 




Thus, from the point of view of designing a sample for 
estimating a single item parameter, the case of all examinee 
parameters equal to the (unknown) item parameter would be 
ideal . 

A slightly more complicated problem is che one of 
designing an optimal sample for the simultaneous estimation 
of I iten parameters with possibly different values. Then for 
each item parameter each examinee has a different 
contribution to the information in the sample. A moment's 
reflection reveals that now the optimal distribution of 
examinees in the sample is a function of the desired 
distribution of information over the item parameters. Hence, 
this distribution has to be specified before the sample can 
be designed. The following model gives a solution to the 
problem. Let r^ be the relative amount of information about 
item parameter 5^ wanted from the sample. Let xk be the 
unknown sample - frequency of examinees with parameter value 

9k» k = 1 where the grouping of examinees can be 

a.ade arbitrarily fine. Finally, suppose the response 
variables A^ j of different examinees are independent . Then , 
because of the additivity of Fisher's measure, the optimal 
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distribution of examinees is the value of (xj x K ) 

solving the following integer programming problem: 



(4) maximize Z k=1 I(8 io ;6 k )xk 



subject to 



(5) ri 1 Sj =1 I(S 1 ;0 k )xk = ... = r" 1 l£ =1 I(S I ;e k )xk. 



(O lj =1 xk = J. 



(7) , x k = 0, 1 7, k = 1 K. 

The idea underlying this formalization is to maximize the 
information for an arbitrary item i 0 under the condition that 
the distribution of information over all item parameters must 
be proportional to (r^, .... rj). As a consequence, t^e 
distribution of the J examinees is chosen such that the 
relative distribution of information as specified by (ri ( , 
.... rj) is "blown up as far as possible". Algorithms for 
solving (4) to (7) can be found in the integer programming 
literature (e.g.. Wagner, 1975. chap. 13) and are amply 
available in computer code. 

The above problem was given only as a theoretical 
exercise; in practice , the item parameters are unknown, and 
for finite values of I an exact solution to the equality 
constraint in (5) is unlikely to exist. However, before 

3 
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giving a practical solution based on the same idea, it is 
observed that the problem is complicated somewhat further if 
the assumption that each examinee gets the same set of items 
should be abandoned. Such incomplete samples occui in 
situations where the number of items is too large to 
administer all items to each examinee, for example, when 
large item banks or sets of parallel tests have to be 
constructed. 

It is the purpose of the present paper to give some 
optimization models for incomplete sample designs. For these 
models to be practical, it is necessary that, unlike the 
above example, the item parameters need not be known 
beforehand Several cases are considered. In the first case, 
the assumption of known item parameter values is dropped. The 
primary goal of presenting this case is to introduce the 
methodology and to show how it applies to the present 
problem. In the next case the assumption of known values for 
the ability parameters is dropped as well. The results are 
then generalized: first to the case of multiple ability 
scales and next to the case of models with multiple item 
parameters. An illustrative axample concludes the paper. 
Following the practice of survey sampling (Cassel, Sarndal . & 
Wretman. 197*;). a distinction between stochastic and 
5 deterministic designs is made. All results are formulated as 

linear programming models. Linear programming was applied 
earlier in test theory to the problem of test construction: 
first by Theunissen (1985). and subsequently by Adema (1988). 
Boekkooi-Timminga (1987). Theunissen (1986). Theunissen and 
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Verstralen (1986). van der Linden (1987). van der Linden and 
Adema (1988). and van der Linden and Boekkooi-Timminga (1988, 
in press). A sequential solution to the problem of sample 
optimization not based on a linear programming approach is 
given in van der Linden and Eggen (1986). 

General Shape of the Sample 

Mostly, the design of the sample for calibrating a large item 
bank has to meet several conditions. The following types of 
conditions can be distinguished. 

First, the domain of content for which the items are 
constructed may have a structure of problems nested in 
topics. The topics, in turn, may display a hierarchical 
structure of specific topics nested in broader ones. Also, 
different groups of topics usually require different 
abilities to solve the problems. Although the classification 
of items into topics is only a matter of judgment by experts, 
the ability structure enters the design as an empirical 
hypothesis to be tested on response data. 

Second, the population of examinees usually also has a 
hierarchical structure, with examinees nested in classes, 
schools, districts, and so on. In addition, social or 
geographical factors may play a role. In the practice of item 
calibration, group-based designs are common. They reduce the 
logistic efforts involved in administering the items and 
leave units like school classes intact. 
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Third, various practical conditions may exist, for 
example, with respect to the composition of the tests, 
curricular differences between schools, or the time available 
for item administration . In order to avoid the problem of 
nonresponse, the sample should be designed to meet such 
conditions as well. 

The general shape of the sample considered in the paper 
is given in Figure 1. The examinees in the operational 



Insert Figure 1 about here 



population are denoted by j = 1 J. Each examinee is 

nested in one of the groups g = 1 G. The ability scales 

are denoted by s = 1 , S,, where scale s consists of 

topics t = 1 T s and topic t contains item i = 1 

I t . Additional levels of groups and topics can easily be 
introduced by further partitioning the examinees and the 
items, but are ommitted here for convenience. The design 
variables Xji ts and Xg its represent the decision on examinee 
j and group g with regard to item (i.t.s). Two possible types 
of designs are considered: (1) Stochastic designs , where the 
variables are probabilities of assigning items to examinees 
(or groups). The problem is to optimize the probabilities; 
once they are known, auxiliary experiments can be used to 
decide on the actual administration of the items. Sampling 
with a stochastic design is common in survey research 

e 1,2 
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(Cassel, Sarndal, & Wretman, 1977), but has remained 
unnoticed in IRT ; ( 2 ) Deterministic designs , where the 
variables take only the vales one or zero, indicating whether 
or not the item is administered to tt*e examinee or the group. 

The strategy in the following part of the paper is to 
start with the simple case of one scale and one topic and 
then to generalize to the more complete case in Figure 1 . 
Although the models are formulated either for examinees or 
groups as sampling units, they can easily be transformed to 
the other level . 



Unknown Item Difficulties and Known Abilities 

Suppose we have at o'ur disposal J examinees with known values 
for their ability parameters. Such cases are not frequently 
met in practice, but may arise in a two-stage testing 
strategy where a short pretest is used to optimally aszi^ 
items to examinees in the second stage. 

Since the values of the item parameters are unknown, a 
target for the information in the sample over the scale of 
possible values of the parameter has to be given. For a fixed 

parameterization, let 8 il 8 ic j , 8^ be the possible 

values of parameter for which the target is specified (the 
number of points can be made arbitrarily large). Two types of 
models are presented each based on a different target: (1) A 
relative target, denoted by a vector of nonnegative numbers 

Pil Pi£). These prespecif ied numbers indicate the 

informati m the sample should have on item parameter 8^ if it 

... Etyc *3 
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would take one of the values of 8j^ , . . . , relative to the 

others; (2) An absolute target specifying on a 

fixed scale the actual amount of information the sample 

should have for each of the values 8^ 8^. The former 

only specifies the shape of the distribution of information 
(e.c,\, uniform, skewed to the left, etc.); the latter also 
specifies its size. The relative target is used in the 
following model (cf. van der Linden, 1986). 

Model of Maximal Information 

It is assumed that the sample size is fixed and that the 
objective is to maximize the information in the sample such 
that the shape of the target distribution is realized 
maximizing its size. Formally, this can be achieved by 
replacing (Pn. Pip) by the vector of products (p^y, 

. ... PiDy) • where y is a scale factor to be maximized. 
The model is: 

(8) maximize y 
subject to 

(9) 2j b1 I<8 id ;e.j)xji - p id y > 0, i = 1 I, 

d = 1 D; 

(10) E* sl xji < nj, 3 = 1 J; 
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(11) 0 S Xji s 1, 1=1 I. 

3 = 1 J; 

(12) y 0. 

In (8). y is maximized so that the lower bounds to 
1(5^ ;8j )xji in (9) are maximal while still obeying the 

ratio's specified by (pjj PiD>- The length of the test 

for examinee j is restricted to n-j in (10), and the 
constraints in (11) specify a design with stochastic 
variables. The model is linear in the design variables and 
can be solved by the simplex algorithm (e.g., Wagner. 1975, 
chap. 4). Also, for any set of constants > 0 and nj > 0 a 
feasible solution exists. This property has been achieved by 
replacing the equality constraints in (5) by the inequalities 
in (9). Further, because thcf lower bounds in (9) are 
maximized, the solution hardly exceeds these bounds. 
Therefore, in practice, (9) works as an equality constraint. 
For the same reason, the expected test length will reach the 
upper bound in (10). 

It should be noted that the constants p^ and nj allow 
specification of the results at the level of individual items 
and examinees. This is the most general formulation of the 
model. However, in practice the target for the information in 
the sample will be the same for all items, whereas the 
required test length will be the same for groups of 
examinees. This reduces the number of coefficients to be 
specified in the model. 
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Unknown Item Difficulties and Unknown Abilities 

Suppose now the values of the ability parameters are not 
known . In such cases , however , some kind of prior knowledge 
about the distribution of the abilities in groups of 
examinees may be available. For example, if a new item bank 
has to be calibrated , norms for the old bank may be known ; 
or. perhaps, in a national assessment study of educational 
achievements results from a previous study could be used. 

Such information can be exploited to optimize the 
sample, provided one of the old items is added to the sample 
to equate the scale for the new items to the old one. 
Although the following models are formulated for empirical 
priors at group level, the same type of models can be 
formulated for subjective priors at the level of individual 
examinees if one is ready to accept the assumption of 
exchangeability of examinees. 

Model of Maximal Expected Information 

Let TgCO) be the prior distribution for the population from 
which group g can be sampled with size Ng. The expected 
information on item parameter 8 in group g is equal to 



(13) 
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The model of maximal expected information follows on 
substituting (13) for 1(5^ ;8) in (8) through (12), replacing 
the design variables Xj^ by Xg^. 

flodel of Minimal S ampl e Size 

If the objective is not to maximize the expected information, 
but to minimize the sample size for an absolute target, 
additional decision variables are needed to include group g 
in the sample: 



(14) Zg 2 



1 group g in the sample 

0 otherwise g = 1 G. 



The model If as follows : 

(15) minimize ^ 
subject to 

(16) SjLj Eg[I(8 id ;0)]xgi > a id . i = 1, .... I. 

d = 1 D;. 

(17) n^Zg - Z* =1 xgi < 0. g = 1 G; 

(18) l\ sl xgi - n g 2) z g < 0. g = 1 G- 
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(19) Z g , X gi € {0, 1}, i = 1 I, 

g = l G. 

The objective function in (15 ) minimizes the number of 
groups, while (16) guarantees that the information in the 
sample is not below target. The latter also implies that each 
item is assigned to at least one group. The constraints in 

(17) and (18) serve a double goal: they set lower and 

CI) r 2) 

upper bounds n£ 'and n£ to the length of the test for group 
g at the same time preventing the possibility of excluding g 
from the sample while assigning items to it. Because the 
objective is to minimize the sample size, for most groups 
only the upper bound in (18) is effective. In (19), 
deterministic design variables are specified. For this type 
of variable, a branch-and-bound algorithm can be used to 
solve the model (e.g., Wagner, 1975, chap. 13). Adema (1988) 
gives a modified branch-and-bound procedure that solves 
large-scale zero-one programming problems in realistic time. 
If the design variables are chosen to be stochastic. (17) and 

(18) set lower and uppei bounds to the expected test length. 
When specifying n^ 1 * , it should hold, of course, that 

2^ =1 £ I. If this condition is satisfied, the model has 

a feasible solution if the target is set not too high 
relative to the maximum of zg . 
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Multiple Ability Scales 

The above models are also appropriate for the case of 
multiple ability scales . 

Suppose that S abilities are necessary to cover the 
complete set of items, and that each item requires exactly 
one of these abilities. Further, for each of the items in set 

s = 1 S the response model in (1) holds, but separate 

likelihoods are necessary to estimate the item parameters in 
different sets. Let I s denote the set of indices of the items 
in s. while ability 8 S is needed to solve an item in s. 

The only thing necessary to apply the above models to 
the case of multiple abilities is to take account of scale 
differences between the item sets. For the case of known 
abilities, the item parameter scales in the different sets of 
items are automatically fixed by their ability scales. If 
empirical priors are used, the scales of the item parameters 
have to be equated to the tests from which the priors were 
extracted. As already noted, this can be done by adding for 
each set rn old item to the sample. 

Suppose the interest is in the group-based model for 
maximal expected information, that is. in (8) through (12) 
with (9) replaced by (13). It should be noted that for this 
model, the effect of one numerical unit of the objective 
function depends on the item parameter scales chosen. This 
should be taken into account when specifying the targets for 
the separate item sets relative to each other. Let p^, 
d = 1 D, be the target for item i € I s . s = 1, .... S, 

ERIC i 9 
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and let rg S (8 s ) be the prior for ability 8 S in group g. The 
model is now: 

(20) maximize y 



subject to 



(2i) 2^ =1 {N g \ i(5 id; e s )dr gs (e s )])xgi - Pidy > o. 

i € Is . 
s = 1, 

d = 1, 



. . . S, 
. . . D; 



(22) 



^1 

L i=1 Xgi < ng. 



g = 1, • • . . G; 



(23) 0 < x gi < 1, i a 1. .... I. 

s = 1 S. 

g = 1. . . . . G; 



(24) y > 0. 



Multiple Item Parameters 



So far, the Rasch model in (1) has been assumed to explain 
the probabilities of correct item responses. However, the 
only role it played was in the coefficients of constraints 
like (9), (13), (16), or ^21); the structure of the 
optimization models was independent of the specific form of 

ERIC 20 
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the IRT model. In fact, only the number of item parameters 
counts, and the above models can be used for any one- 
parameter IRT model. The question arises how these models 
look like for IRT models with more than one item parameter. 

For the sake of illustration, the two-parameter logistic 
model is adopted: 

(25) Pi(6j) = {1 + exp [ai(bi-8j)] J"" 1 , 

with -oo < b^, 8j < +oo and ai > 0, where ai and bi are the 
discriminating power and difficulty of item i, respectively 
(e.g., Hambleton & Swaminathan, 1935, sect. 3.3.2; Lord. 
1980). For Fisher's information measure, it follows from (25) 
that 

(26) I*(7t i ;e 1 0j) 

= £? =1 Opi(e j )/aK i )2{ Pi (e j )[i- P i(9 j ))) ~\ 

x± = ai, bi. 

The problem is to design the sample simultaneously with 
respect to information on parameters a^ and b^ . How this can 
be achieved by a simple reinterpretation of the above models 
is shown for uhe model in (15) through (19). 

Let a il a iv a iV ^d b n b iw b iW 

be the possible values of parameters a^ and b^ for which 
absolute target values for the expected information are 

ERLC 21 
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specified. As in (26), let n i run over (aj., bi); also let p 
run over (v, w) . The model in (15) through (19) generalizes 
to the case of two item parameters, if (16) is replaced by 

(27) Eg[I*(7C ip; G)]Xgi > CX ip , i = 1. I. 

p = v, w, 

v = 1 V, 

w = 1 W. 

The only difference is the doubling of the number of 
constraints in (27) and a change of coefficients; the 
structure of the model remains the same. 

Again, it should be noted that the targets for different 
parameters are set on different scales. If the difficulty 
scale is fixed by known abilities or the use of a known prior 
distribution, the scale of the discriminating power 
parameters is also fixed, but with reciprocal unit. 

From the above, the generalization to more than two item 
parameters is obvious . 

Practical Conditions on the Tests 

As noted earlier, several practical conditions on the tests 
for the various groups in the sample design may be in force. 
Such conditions can be inserted in the optimization models 
given here without destroying their properties, provided they 
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can be formulated as linear (in)equalities , Some examples are 
given. 

Test content . If for some groups of examinees the number 
of items on topic t are not allowed to be larger than ng t , 
constraints like (10) should be replaced by 

(28) X xgi < ngt. for some g and t, 
i€l t 

Because in the models for maximal information (28) is likely 

to work as an equality constraint, the composition of the 
T 

test is fixed if ngt = ng. 

Curricular fit. Suppose different groups have 

experienced different curricula with respect to the domain of 
content covered by the test items. If topic t has not been 
taught to group g. its items can be suppressed by setting rig t 
a 0 in (28). In addition, it is possible to suppress 
individual items by 

(29) x gi =0, for some i. 

Test security . For reasons of test security it may be 
desirable to prevent some items from administering to more 
than v groups. This is achieved by the following linear 
constraint : 



(30) 




for some i . 
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Simultaneous inclusion of groups . If groups with 
different priors belong to the same organization and a model 
of minimal sample size is chosen, it may be useful to make 
the same decision about inclusion in the sample for these 
groups : 

(31) Zg = Zg* , for some (g, g'). 

However, the same effect can be obtained by replacing the 
priors for such groups by mixtures. 

The above constraiats are only a small selection from 
the possibilities. In principle, the same constraints may be 
needed as when designing customized tests from a calibrated 
item bank. A more complete treatment of the latter is given 
in van der Linden and Boekkooi-Timminga (in press). 

Example 

At the end of the seventies, the Dutch part of the Second 
Mathematics Study of the International Association for the 
Evaluation of Educational Achievement (IEA) was conducted. 
For several subjects the mathematics achievements of grade- 
seven pupils were assessed and compared with results in other 
countries. Suppose a third study has been planned and the 
methodology in this paper has to be used to optimize the 
sampling design for calibrating the test items As model for 
the item responses, the Rasch model in (1) is chosen. Three 
different subject areas are distinguished (Geometry, Algebra, 

24 
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and Arithmetic), and achievements in three different types of 
secondary education are to be reported (LTE: Lower Technical 
Education; DSE: Domestic Science Education; MGE: Middle 
General Education) . 

For a sample of 400 examinees from the Second 
Mathematics Study the Rasch model with a normal distribution 
over the ability parameters was fitted to the data. The 
results are given in Table 1. The numbers of items were equal 



Insert Table 1 about here 



to 10 (Geometry). 9 (Algebra), and 11 (Arithmetic), All 
estimates of \i and a were transformed to the same scale, 
which was obtained by fitting the model to the total set of 
items for the LTO examinees (p = .428). 

The normal distributions in Table 1 were used as 
empirical priors in the model of maximal expected information 
in (8) through (13) with (9) replaced by (13). The sampling 
plan used in this example is given in Table 2. Since no items 



Insert Table 2 about here 




were assumed to deserve a special treatment, the same uniform 
target was set for all items (i.e., p ic j =1, i = 1 I, 
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d = 1 D) . The estimated difficulties of the previous 

items varied between -.852 and 2.010; therefore, the target 
was set at the points 8^ = — .50. ?2 = +-50. and 83 - +1.50. 
The design variables in the model were allowed to take values 
between zero and one. 

The model was solved using a standard computer program 
for linear programming problems on a DEC2060 system (LANDO: 
Anthonisse . 1984) . The result, which is just an array of 
numerical values for the 900 design variables, is not shown 
here. For future reference, it is only noted that 831 
variables were equal to zero or one whereas 69 variables took 
values between zero and one. The CPU time needed was 28 mins. 
and 27 sees. In addition, the value of the objective function 
for the solution was obtained. It was equal to 71.228. Since 
this is a lower bound to Fisher's information, it follows 
that, uniformly in 6, an asymptotic standard error of 
estimate not larger than (71 .228)*/2 = n% can ^ e expected. 

Final Remarks 

For the sampxe design in the above problem, the actual 
assignment of items to examinees could take place using a 
random experiment. However, other options are available. 

First, the design variables in the model could be 
replaced by zero-one variables . Since the number of variables 
in the example generally is too large for a branch-and~- bound 
algorithm, a heuristic as in Adema (1988) could be used. The 
heuristic is based on the principle that, for a maximization 
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problem, the value of the objective function for a solution 
with relaxed variables is an upper bound to the one for the 
zero-one problem. Hence, as the final solution it accepts the 
first feasible solution with an objective function value 
differing no more than a small percentage , 0,5%, say. from 
the upper bound. The heuristic has proven to solve large- 
scale problems in realistic time ( Adema. 1988 ) . 

Second, for larger numbers of decision variables than in 
the example, splxtting a zero-one problem into smaller 
problems for which a branch-and-bound algorithm could be used 
is an attractive option. In the example, the design matrix 
could be partitioned along its rows, computing an optimal 
solution for subsets of groups. In this option, the model has 
not to be adapted but for the range of the group index. If an 
absolute target were to be used, targets of subproblems 
should allow for possible differences in numbers cf subjects. 

The final option for a model as in the example is 
rounding the solution of the relaxed problem. It is known 
that the number of decision variables in the solution with 
noninteger values is not larger than the number of 
constraints (Dantzig. 1957); in practice, it is considerably 
smaller. The number of noninteger values in the example was 
equal to 69. These values can be rounded optimally by a zero- 
one programming problem consisting of the previous problem 
with the other variables fixed at their values in the 
previous solution. The objective now is to approach the 
information on the item parameters in the this solution as 
close as possible. 
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Table 1 



Parameter estimates of 


the 


empirical 


priors for 


different 


types of schools 


and subject 


areas . 






Subject Parameter 






Type of School 


Area 






.LTE 


.DSE 


.LGE 




H 




-.446 


-1.263 


.053 


Geometry 


a 

pi) 




.809 
.529 


.665 
.577 


.920 
.749 




p*2) 




.658 


.237 


.499 




H 




.446 


-.440 


1.657 


Algebra 


a 
P 




.700 
.423 


.704 
.670 


.950 
.145 




P 




.056 


.474 


.871 




H 






-.690 


.649 


Arithmetic 


a 
P 




.950 
.344 


.512 
.223 


.888 
.256 




P 




.962 


.666 


.792 


Notes : 1) probability 


of 


exceeding 


the result 


for the 


1 ikel ihood-r at io 


test ( 


on 


the Rasch 


model; 2) 


idem for 



assumption of normality. 
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Table 2 

Numbers of groups and items and group sizes in the sample. 

Types of No. of Group Subject No. of 

School Groups Size Area Items 



LTE 4 32 Geometry 25 

DSE 6 35 Algebra 15 

MGE 5 28 Arithmetic 20 
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Scale 1 ... s 

Topic 1 ... t ... T s 

Item 1 ... i ... It- 



Group Examinee 
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1 



g 



x jits 
x gits 



Figure L General shape of a sample for estimating IRT 

parameters 
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